{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Analysis of hyperparameter search results"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the previous notebook we showed how to implement a randomized search for\n",
    "tuning the hyperparameters of a `HistGradientBoostingClassifier` to fit the\n",
    "`adult_census` dataset. In practice, a randomized hyperparameter search is\n",
    "usually run with a large number of iterations."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In order to avoid the computational cost and still make a decent analysis, we\n",
    "load the results obtained from a similar search with 500 iterations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "\n",
    "cv_results = pd.read_csv(\n",
    "    \"../figures/randomized_search_results.csv\", index_col=0\n",
    ")\n",
    "cv_results"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lines_to_next_cell": 2
   },
   "source": [
    "We define a function to remove the prefixes in the hyperparameters column\n",
    "names."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def shorten_param(param_name):\n",
    "    if \"__\" in param_name:\n",
    "        return param_name.rsplit(\"__\", 1)[1]\n",
    "    return param_name\n",
    "\n",
    "\n",
    "cv_results = cv_results.rename(shorten_param, axis=1)\n",
    "cv_results"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we have more than 2 parameters in our randomized-search, we cannot\n",
    "visualize the results using a heatmap. We could still do it pair-wise, but\n",
    "having a two-dimensional projection of a multi-dimensional problem can lead to\n",
    "a wrong interpretation of the scores."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import seaborn as sns\n",
    "import numpy as np\n",
    "\n",
    "df = pd.DataFrame(\n",
    "    {\n",
    "        \"max_leaf_nodes\": cv_results[\"max_leaf_nodes\"],\n",
    "        \"learning_rate\": cv_results[\"learning_rate\"],\n",
    "        \"score_bin\": pd.cut(\n",
    "            cv_results[\"mean_test_score\"], bins=np.linspace(0.5, 1.0, 6)\n",
    "        ),\n",
    "    }\n",
    ")\n",
    "sns.set_palette(\"YlGnBu_r\")\n",
    "ax = sns.scatterplot(\n",
    "    data=df,\n",
    "    x=\"max_leaf_nodes\",\n",
    "    y=\"learning_rate\",\n",
    "    hue=\"score_bin\",\n",
    "    s=50,\n",
    "    color=\"k\",\n",
    "    edgecolor=None,\n",
    ")\n",
    "ax.set_xscale(\"log\")\n",
    "ax.set_yscale(\"log\")\n",
    "\n",
    "_ = ax.legend(\n",
    "    title=\"mean_test_score\", loc=\"center left\", bbox_to_anchor=(1, 0.5)\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the previous plot we see that the top performing values are located in a\n",
    "band of learning rate between 0.01 and 1.0, but we have no control in how the\n",
    "other hyperparameters interact with such values for the learning rate.\n",
    "Instead, we can visualize all the hyperparameters at the same time using a\n",
    "parallel coordinates plot."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import plotly.express as px\n",
    "\n",
    "fig = px.parallel_coordinates(\n",
    "    cv_results.rename(shorten_param, axis=1).apply(\n",
    "        {\n",
    "            \"learning_rate\": np.log10,\n",
    "            \"max_leaf_nodes\": np.log2,\n",
    "            \"max_bins\": np.log2,\n",
    "            \"min_samples_leaf\": np.log10,\n",
    "            \"l2_regularization\": np.log10,\n",
    "            \"mean_test_score\": lambda x: x,\n",
    "        }\n",
    "    ),\n",
    "    color=\"mean_test_score\",\n",
    "    color_continuous_scale=px.colors.sequential.Viridis,\n",
    ")\n",
    "fig.show(renderer=\"notebook\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"admonition note alert alert-info\">\n",
    "<p class=\"first admonition-title\" style=\"font-weight: bold;\">Note</p>\n",
    "<p class=\"last\">We <strong>transformed most axis values by taking a log10 or log2</strong> to\n",
    "spread the active ranges and improve the readability of the plot.</p>\n",
    "</div>\n",
    "\n",
    "The parallel coordinates plot displays the values of the hyperparameters on\n",
    "different columns while the performance metric is color coded. Thus, we are\n",
    "able to quickly inspect if there is a range of hyperparameters which is\n",
    "working or not.\n",
    "\n",
    "It is possible to **select a range of results by clicking and holding on any\n",
    "axis** of the parallel coordinate plot. You can then slide (move) the range\n",
    "selection and cross two selections to see the intersections. You can undo a\n",
    "selection by clicking once again on the same axis.\n",
    "\n",
    "In particular for this hyperparameter search, it is interesting to confirm\n",
    "that the yellow lines (top performing models) all reach intermediate values\n",
    "for the learning rate, that is, tick values between -2 and 0 which correspond\n",
    "to learning rate values of 0.01 to 1.0 once we invert back the log10 transform\n",
    "for that axis.\n",
    "\n",
    "But now we can also observe that it is not possible to select the highest\n",
    "performing models by selecting lines of on the `max_bins` axis with tick\n",
    "values between 1 and 3.\n",
    "\n",
    "The other hyperparameters are not very sensitive. We can check that if we\n",
    "select the `learning_rate` axis tick values between -1.5 and -0.5 and\n",
    "`max_bins` tick values between 5 and 8, we always select top performing\n",
    "models, whatever the values of the other hyperparameters."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this notebook, we saw how to interactively explore the results of a large\n",
    "randomized search with multiple interacting hyperparameters. In particular we\n",
    "observed that some hyperparameters have very little impact on the\n",
    "cross-validation score, while others have to be adjusted within a specific\n",
    "range to get models with good predictive accuracy."
   ]
  }
 ],
 "metadata": {
  "jupytext": {
   "main_language": "python"
  },
  "kernelspec": {
   "display_name": "Python 3",
   "name": "python3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}